* wip

* more

* Make scalar.rs compile

* Fix various compilation error due to API difference

* Make datafusion core compile

* fmt

* wip

# This is the 1st commit message:

Add Display for Expr::BinaryExpr

# This is the commit message #2:

Update logical_plan/operators tests

# This is the commit message #3:

rebase and debug display for non binary expr

* # This is a combination of 3 commits.
# This is the 1st commit message:

Add Display for Expr::BinaryExpr

# This is the commit message #2:

Update logical_plan/operators tests

# This is the commit message #3:

rebase and debug display for non binary expr

* Add Display for Expr::BinaryExpr

Update logical_plan/operators tests

rebase and debug display for non binary expr

Add Display for Expr::BinaryExpr

Update logical_plan/operators tests

Updating tests

Update aggregate display

Updating tests without aggregate

More tests

Working on agg/scalar functions

Fix binary_expr in create_name function and attendant tests

More tests

More tests

Doc tests

Rebase and update new tests

* Submodule update

* Restore submodule references from master

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* Initial commit

* initial commit

* failing test

* table scan projection

* closer

* test passes, with some hacks

* use DataFrame (#2)

* update README

* update dependency

* code cleanup (#3)

* Add support for Filter operator and BinaryOp expressions (#4)

* GitHub action (#5)

* Split code into producer and consumer modules (#6)

* Support more functions and scalar types (#7)

* Use substrait 0.1 and datafusion 8.0 (#8)

* use substrait 0.1

* use datafusion 8.0

* update datafusion to 10.0 and substrait to 0.2 (#11)

* Add basic join support (#12)

* Added fetch support (#23)

Added fetch to consumer

Added limit to producer

Added unit tests for limit

Added roundtrip_fill_none() for testing when None input can be converted to 0

Update src/consumer.rs

Co-authored-by: Andy Grove <andygrove73@gmail.com>

Co-authored-by: Andy Grove <andygrove73@gmail.com>

* Upgrade to DataFusion 13.0.0 (#25)

* Add sort consumer and producer (#24)

Add consumer

Add producer and test

Modified error string

* Add serializer/deserializer (#26)

* Add plan and function extension support (#27)

* Add plan and function extension support

* Removed unwraps

* Implement GROUP BY (#28)

* Add consumer, producer and tests for aggregate relation

Change function extension registration from absolute to relative anchor
(reference)

Remove operator to/from reference

* Fixed function registration bug

* Add test

* Addressed PR comments

* Changed field reference from mask to direct reference (#29)

* Changed field reference from masked reference to direct reference

* Handle unsupported case (struct with child)

* Handle SubqueryAlias (#30)

Fixed aggregate function register bug

* Add support for SELECT DISTINCT (#31)

Add test case

* Implement BETWEEN (#32)

* Add case (#33)

* Implement CASE WHEN

* Add more case to test

* Addressed comments

* feat: support explicit catalog/schema names in ReadRel (#34)

* feat: support explicit catalog/schema names in ReadRel

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: use re-exported expr crate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* move files to subfolder

* RAT

* remove rust.yaml

* revert .gitignore changes

* tomlfmt

* tomlfmt

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Daniël Heres <danielheres@gmail.com>
Co-authored-by: JanKaul <jankaul@mailbox.org>
Co-authored-by: nseekhao <37189615+nseekhao@users.noreply.github.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>

* add mut annotation

* fix rust examples

* fix rust examples

* update

* fix first doctest

* fix first doctest

* fix more doctest

* fix more doctest

* fix more doctest

* adopt rustdoc syntax

* adopt rustdoc syntax

* adopt rustdoc syntax

* fix more doctest

* add missing imports

* final udtf

* reenable

* remove dep

* run prettier

* api-health

* update doc

* update doc

* temp fix

* fix doc

* fix async schema provider

* fix async schema provider

* fix doc

* fix doc

* reorder

* refactor

* s

* finish

* minor update

* add missing docs

* add deps (#3)

* fix doctest

* update doc

* fix doctest

* fix doctest

* tweak showkeys

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* update to use user_doc

* add rustdoc preprocessing

* fix dir

* revert to original doc

* add allocator

* mark type

* update

* fix doctest

* add doctest

* add doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* fix doctest

* prettier format

* revert change to datafusion-testing

* add apache header

* install cmake in setup-builder for ci workflow dependency

* taplo + fix snmalloc

* Update function docs

* preprocess user-guide

* Render examples as sql

* fix intro

* fix docs via script

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

… default null ordering (#16963)

* Allow to set up the default null ordering (#3)

* add default_null_ordering config

* add test for different config

* Update datafusion/sql/src/planner.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update doc

* fix sqllogictest

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* rename config and fix test

* update doc

* fix default

* fix doc

* fix sqllogictests

* address comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

… default null ordering (apache#16963)

* Allow to set up the default null ordering (apache#3)

* add default_null_ordering config

* add test for different config

* Update datafusion/sql/src/planner.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* update doc

* fix sqllogictest

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* rename config and fix test

* update doc

* fix default

* fix doc

* fix sqllogictests

* address comments

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update MD

* Update asf.yaml

* Cleaner name

* keep protected branch

* Remove protected branches for now

* Put `protected_branches` back

* Remove all protected branches

* Remove website + publish

## Which issue does this PR close?

- Part of apache#23334.

> The numbers below come from the committed criterion benchmark added in
apache#23335 (`cargo bench --bench
array_has`) — **origin** = the per-row `eq` kernel (unoptimized `main` /
apache#23335), **now** = with this
optimization applied. Run the bench on `main` and on this branch to
reproduce.

Full disclosure - this was heavily assisted by AI, and I did my best to
understand and justify every change here before submitting.

## Rationale for this change

`array_has(array, element)` returns, for each row, whether the array
contains the element.

When the `element` (needle) is an array rather than a scalar, the needle
argument is a column with one value per row, e.g. `array_has(t1.tags,
t2.key)` in a join filter, execution goes through
`array_has_dispatch_for_array` (the `ColumnarValue::Array` needle
branch), which compared each row by invoking the Arrow `eq` kernel once
per row.

That kernel allocates a `BooleanArray` and pays downcast and dispatch
overhead on every row. (The scalar-needle branch was optimized
separately in apache#20374.)

What this removes is the fixed per-row kernel overhead, not the element
comparison itself, so the gain is largest for short lists and shrinks as
lists grow.

All numbers below are from the committed criterion benchmark (`cargo
bench --bench array_has`, groups `array_has_array_null_patterns` /
`array_has_array_by_size` / `array_has_array_by_rows`): the `array_has`
UDF evaluated in isolation with an array needle, **origin** (the per-row
`eq` kernel) vs **now**. "list length" is the number of elements in each
row's array (not the row count). Not end-to-end query time.

### By data type and null pattern (list length 64, 10K rows)

| element | element len | null pattern | origin | now | speedup |

|-----------|----------------|----------------------|---------|---------|---------|
| i64 | - | no nulls, found | 1.10 ms | 73 µs | 15.1x |
| i64 | - | no nulls, not found | 1.07 ms | 72 µs. | 14.9x |
| i64 | - | 30% nulls, found | 1.17 ms | 315 µs | 3.7x |
| i64 | - | 30% nulls, not found | 1.10 ms | 274 µs | 4.0x |
| i64 | - | all null | 1.10 ms | 272 µs | 4.0x |
| i64 | - | collision | 1.10 ms | 270 µs | 4.1x |
| Utf8 | short (inline) | no nulls | 2.57 ms | 1.01 ms | 2.5x |
| Utf8 | short (inline) | 30% nulls | 3.37 ms | 1.52 ms | 2.2x |
| Utf8 | long (>12B) | no nulls | 2.61 ms | 1.04 ms | 2.5x |
| Utf8 | long (>12B) | 30% nulls | 3.31 ms | 1.52 ms | 2.2x |
| Utf8 | - | all null | 1.26 ms | 256 µs | 4.9x |
| LargeUtf8 | short (inline) | no nulls | 2.56 ms | 1.02 ms | 2.5x |
| LargeUtf8 | short (inline) | 30% nulls | 3.20 ms | 1.54 ms | 2.1x |
| LargeUtf8 | long (>12B) | no nulls | 2.67 ms | 1.05 ms | 2.6x |
| LargeUtf8 | long (>12B) | 30% nulls | 3.42 ms | 1.59 ms | 2.2x |
| LargeUtf8 | - | all null | 1.31 ms | 263 µs | 5.0x |
| Utf8View | short (inline) | no nulls | 1.18 ms | 239 µs | 4.9x |
| Utf8View | short (inline) | 30% nulls | 1.26 ms | 246 µs | 5.1x |
| Utf8View | long (>12B) | no nulls | 2.86 ms | 1.17 ms | 2.4x |
| Utf8View | long (>12B) | 30% nulls | 3.51 ms | 1.66 ms | 2.1x |
| Utf8View | - | all null | 1.20 ms | 267 µs | 4.5x |

The i64 null cases are uniform (~4x) whether the match is present,
absent, the whole list is null, or the needle collides with a null
slot's backing fill value — validity is folded in with one word-parallel
op, so there is no per-row rescan and no null slot can match.

Strings win ~2.1–2.5x mainly by dropping the per-row `BooleanArray`
allocation. `Utf8View` additionally uses a view-aware compare: the byte
length and 4-byte prefix packed into the 128-bit view reject non-matches
before touching the data buffer, and an inline value (≤ 12 bytes) is
matched by whole-view equality with no materialization at all — hence
~5x on short/inline strings. When long strings share a prefix (e.g.
ARNs) the prefix can't reject, so `Utf8View` falls in line with the
other string types (~2.1–2.4x). No string case regresses.

### By list length (i64, 30% element nulls, not found, 10K rows)

| elems/row | origin  | now     | speedup                              |
|-----------|---------|---------|--------------------------------------|
| 8         | 1.03 ms | 111 µs  | 9.3x                                 |
| 32        | 1.07 ms | 197 µs  | 5.5x                                 |
| 128       | 1.18 ms | 446 µs  | 2.6x                                 |
| 256       | 1.28 ms | 780 µs  | 1.6x                                 |
| 512       | 1.54 ms | 1.44 ms | 1.1x                                 |
| 1024      | 2.17 ms | 2.15 ms | 1.0x (falls back to per-row kernel)  |

The element-null branch makes a few passes over the values; past a
moderate average list length (`NULL_FAST_PATH_MAX_LEN`) the per-row
kernel wins, so it bails to it there — no meaningful regression. That
average is measured over the visible (sliced) region, so a sliced
array's hidden child elements can't route a small window to the slow
path. The all-valid fold has no such crossover.

### By row count (i64, 8 elems/row, 30% nulls, not found)

| rows | origin    | now      | speedup |
|------|-----------|----------|---------|
| 10K  | 1.04 ms   | 111 µs   | 9.4x    |
| 100K | 10.42 ms  | 1.09 ms  | 9.6x    |
| 1M   | 102.68 ms | 10.91 ms | 9.4x    |

Invariant to the number of rows — the per-row overhead removed is a
fixed cost, so absolute savings scale linearly with the column height.

The remaining benchmarks in the suite (scalar `array_has`,
`array_has_all`, `array_has_any` — paths this PR does not touch) are
unchanged (median 0.99x, within measurement noise), confirming no
regression outside the array-needle path.

### End-to-end (context)

For a query dominated by an array-needle `array_has` join filter (a
`NestedLoopJoinExec` with `filter=array_has(tags, key)` over 3000x3000
rows of 8-element lists) total time drops from 0.95s to 0.059s (~16x,
identical results). For a workload where `array_has` is a smaller
fraction, e.g. the ~6% of profile that motivated this (see apache#18070 /
apache#18161, which fixed the join's deep-copy but left the per-row
`array_has` cost), the overall speedup is single-digit percent.

## What changes are included in this PR?

A fast path for primitive and string element types in
`array_has_dispatch_for_array`, preserving the Arrow `eq` kernel
semantics (total-order float equality; null elements never match):

- **All-valid elements:** each row is a single branchless OR-reduction
over the raw native value slice (auto-vectorizes; the common case).
- **Element nulls:** a null slot's backing value is arbitrary, so the
per-element equality bitmap is ANDed with the validity bitmap (one
word-parallel op, no per-element branch) before reducing each row to
"any bit set", a null slot can never match regardless of its value. This
branch is processed in row chunks so the scratch buffer stays bounded,
and past `NULL_FAST_PATH_MAX_LEN` average elements/row a length check
over the visible (sliced) region bails to the per-row kernel (see the
list-length table).
- **String elements:** each row is a single pass over the row's values
(compare, then consult validity only on a match). `Utf8View` compares
the packed 128-bit views directly — length + 4-byte prefix reject
non-matches before any data-buffer access, and an inline value (≤ 12
bytes) matches by whole-view equality with no materialization.
- **Nested (and any other) element types** keep using the per-row `eq`
kernel.

The array-needle benchmarks used for the numbers above are added in apache#3
(null patterns, list length, and row count).

## Are these changes tested?

Yes:

- New unit tests for the array-needle path covering element nulls, the
null-fill collision (needle equal to a null slot's backing value),
total-order float equality (`NaN` / `-0.0`), sliced arrays (including a
small visible window over a large backing child), `LargeList` offsets,
empty rows, a multi-chunk input, and a long-list input that exercises
the per-row fallback, each cross-checked against the original per-row
`eq` kernel as an oracle.
- Existing `array_has` / `array_contains` / `join_lists` sqllogictest
suites pass.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>